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ABSTRACT 



NovaNET is an online computerized instructional system that 
provides students with self-paced instruction for many North Carolina high 
school courses. This evaluation looks at outcomes for students participating 
in the NovaNET program and compares them with outcomes for a group of 
students with similar characteristics who did not participate in NovaNET. The 
comparison group could not be matched exactly with respect to grade point 
average (GPA) ; therefore, some caution should be used in interpreting 
results. The primary positive finding was that the number of failing grades 
decreased significantly for students who particpated in NovaNET, while a 
slight increase occurred for matched students. The GPA for NovaNET students 
showed a significant increase, but the comparison group increased about the 
same amount (differences between increases were not significant) . NovaNET 
appeared to have more of a positive impact on mathematics and social studies 
course grades than on English and science course grades. The pass rate of 
NovaNET science courses was only 50%. NovaNET did not appear to impact 
suspensions in the 2000-2001 school year; rates for NovaNET and matched 
students were statistically similar. NovaNET students were more likely to 
drop out of school in the 2000-2001 school year than the comparison students 
who had a lower dropout rate than WCPSS overall. The change in dropout rates 
should be examined again over the next few years. It is possible that NovaNET 
participants were different from students in the comparison rate in important 
ways. Overall, NovaNET seems to help students pass classes. However, more 
specific findings are difficult to obtain because of the newness of the 
program and the difficulty of creating an accurately matched comparison 
group. In addition, the cost of the program should be considered when 
weighing the benefits. (Author/SLD) 
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ABSTRACT 

NovaNET is an on-line computerized instructional system that provides students with self-paced 
instruction for many North Carolina high school courses. This evaluation looks at outcomes for 
students participating in the NovaNET program and compares them with outcomes for a group of 
students with similar characteristics who did not participate in NovaNET. The comparison group 
could not be matched exactly with respect to grade point average (GPA); therefore , some caution 
should be used in interpreting results. 

The primary positive finding was that the number of failing grades decreased significantly for students 
who participated in NovaNET \ while a slight increase occurred for matched students. The GPA for 
NovaNET students showed a significant increase, but the comparison group increased about the same 
amount (difference between increases were not significant). NovaNET appeared to have more of a 
positive impact on math and social studies course grades than on English and science course grades. 
The pass rate for NovaNET science courses was only 50%. 

NovaNET did not appear to impact suspensions in 2000-2001; rates for NovaNET and matched 
students were statistically similar. NovaNET students were more likely to drop out of school in 2000 - 
2001 than the comparison students who had a lower dropout rate than WCPSS overall; the change in 
drop-out rates should be examined again over the next few years. It is possible that NovaNET 
participants were different from students in the comparison group in important ways. 

Overall, NovaNET seems to help students pass classes. However, more specific findings are difficult 
to obtain due to the newness of the program, and the difficulty of creating an accurately matched 
comparison group. In addition, the cost of the program should be considered when weighing the 
benefits. 
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THE NOVANET PROGRAM 



NovaNET is an interactive on-line learning program through which the curricula for various 
high school courses are taught. The program is self-paced and tests the students’ mastery of 
the subject matter after each unit. NovaNET was first used in the Wake County Public School 
System (WCPSS) on a limited basis by Cary High School and Mary E. Phillips High School 
beginning in the spring of 1996, and was also tried by other alternative schools. In 1999, 
WCPSS received a three-year federal grant called Safe Schools/Healthy Students (SS/HS), 
which provided funding for NovaNET implementation in all Wake County high schools and 
two alternative middle schools. Funding from the SS/HS grant ends September 2002. 
NovaNET is seen as a strategy that has substantial potential to help at-risk high school 
students succeed, but continuation of this program would require considerable resources from 
local or other sources. As a result, this and other evaluation documents were produced to 
allow more informed decisions about future funding for NovaNET. 

An independent study, including surveys and interviews of NovaNET students and staff, was 
done by Faircloth & O’Sullivan (2001). They found that 80% of Wake County NovaNET 
students surveyed reported that working at their own pace was the greatest benefit to 
NovaNET. Many students (38%) also reported that individualized support that NovaNET 
coordinators provided was a benefit of the program. Feeling like an adult at school cares is an 
important asset that lowers students’ risk of dropping out of school. In addition, most 
NovaNET coordinators and guidance counselors (83%) considered NovaNET’s greatest 
benefit to be drop out prevention. 

Three sources exist for information on NovaNET outcomes: 

• Safe Schools/Healthy Students biannual performance reports (“Safe Schools/Healthy 
Students-Year One-End of Year Performance Report,” “Safe Schools/Healthy 
Students-Year Two-Mid-Year Performance Report,” and “Wake County Safe 
Schools/Healthy Students Project — Year Two - Year End Performance Report”) are 
all available on the WCPSS web site at http://www.wcpss.net/evaluation-research/ or 
from E&R. The performance reports address implementation and summarize 
outcomes of all Safe Schools/Healthy Students programs, including NovaNET. 

• “Strategy 17: NovaNET Online Learning System Year 2 Program Evaluation” 
(Faircloth & O’Sullivan, 2001) was produced by an outside researcher. This report 
provides implementation and participation information; in addition, the report includes 
interviews and surveys of both NovaNET students and staff and their views of the 
benefits and drawbacks of NovaNET. 

• This newsletter supplements the other documents by including a comparison group as 
we study the student outcomes for NovaNET. The primary question addressed is 
whether outcomes are better for at-risk students served through this program than for 
similar students who have not participated in NovaNET (which might justify 
additional costs). 
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STUDY DESIGN 



Data Sources 

Information about NovaNET participation has been collected since the inception of the SS/HS 
grant. NovaNET teachers and facilitators at each school reported on the number of students 
taking a NovaNET class, what subject the class covered, and what grade the student achieved 
for the class. In addition, we collected information explaining why the student was referred to 
NovaNET, and whether the student was on track to graduation, closer to graduation but not 
yet on track, or no closer to graduation than at the beginning of the course. 

Demographic and test result data for the NovaNET students were obtained from district 
central computer files (SITS 0625 program). A comparison group of students was chosen 
based on the following criteria: 

• Grade level 

• Gender 

• Race/ethnicity 

• Free/reduced lunch eligibility 

• Eighth grade reading and math End-of-Grade test scores 

• GPA within . 1 

Creating exact matches using these criteria was not possible for all students. As a result, the 
mean initial GPA for the comparison group was significantly higher than for the NovaNET 
group (by 0.39 points). Analyses of grade point averages controlled for this initial difference 
in GPAs, but the groups may differ in other ways that we could not control for statistically 
(attitudes toward school, intrinsic motivation, or parental support, for example). Therefore, 
results for student outcomes must be interpreted cautiously. 

Methods 

In order to assess the effectiveness of NovaNET, we examined outcomes in the following 
ways: 

• We looked at the GPAs of students before and after they participated in the NovaNET 
program. First, we tested for the significance of the increase in GPA with a t-test. 

Then we compared the increase in GPA of NovaNET students to that of the 
comparison group, using a two way analysis of variance (ANOVA). Students were 
only included in these samples if they had a GPA available for both the 1999-2000 and 
2000-2001 school years. Similar statistical tests were used to determine the change in 
number of Fs earned between the two groups, as well as the change in number of 
suspensions, days suspended, and the dropout rate. 

• We also examined the change in GPA for students taking each subject available 
through NovaNET. A t-test was used to measure the increase in GPA for each group, 
although significance was harder to find due to the smaller sample sizes. 
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DEMOGRAPHICS 



Collecting demographic information on the NovaNET participants is important to ensure that 
NovaNET is serving the students for whom it is intended. In the 2000-2001 school year, there 
were 1,213 students participating in the NovaNET program in Wake County Public School 
System (WCPSS). Of those participants, 1,037 students have records available with 
demographic and testing information. Fifty seven of the NovaNET students (5%) were in 8 th 
grade or lower. In addition, 294 NovaNET students (29%) were in 9 th grade, 174 students 
(17%) were in 10 th grade, 181 students (18%) were in 1 1 th grade, and 325 students (32%) 
were in 12 th grade. 

More of the NovaNET participants were male (60%) than female (40%). In addition, Black 
students were over-represented in the NovaNET program (49.5%) compared to WCPSS in • 
general (26.3%), while white students were underrepresented (43.8% vs. 63.6%). The 
percentage of students receiving free or reduced lunch (FRL) was much higher for NovaNET 
participants (51%) than for WCPSS as a whole (19%). Finally, 29% of the NovaNET 
participants were in special education other than Academically Gifted (AG), compared to 
14% in WCPSS. Demographic statistics are visually represented in Figure 1. 

The distribution of NovaNET students’ eighth grade End-of-Grade (EOG) scores provides 
additional evidence that NovaNET is serving at-risk students (see Figure 2). Forty-one 
percent of NovaNET students scored at Level I or II on the reading eighth grade EOG, as 
compared to 13% of all WCPSS students. In addition, 50% of NovaNET participants scored 
at Level I or II on the eighth grade math EOG, compared to 16% of all WCPSS students. 

Thus, students who have at-risk characteristics are disproportionately represented in the 
NovaNET program, suggesting that NovaNET is serving the students whom it was intended 
to reach. 
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Figure 1: Demographic Information of NovaNET Participants Compared to WCPSS 
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Figure 2: Eighth Grade End-of Grade Scores for NovaNET and WCPSS Students 



NovaNET and Non-NovaNET 8th Grade 
Reading EOG Levels 



NovaNET and Non-NovaNET 8th Grade Math 

C I aiiale 




Levels I & II 



Levels lll&IV 



Levels I &II 



Levels III & IV 



RESULTS 



GPA Increase 

To assess the effect that NovaNET had on students’ academic success, we compared 
NovaNET students’ GPA before and after they took the NovaNET course. Data was used 
only for students with available GPA data in both the 1999-2000 and 2000-2001 school year. 
The mean difference between the students’ GPA after NovaNET and before NovaNET is 
0.15. A t-test of this difference shows that the probability of the increase in GPA occurring 
by chance is less than one in 10,000 (T=4.06, p<.0001). 

In addition to measuring the increase in GPA across time, we compared the NovaNET 
students to our comparison group (see “Data Sources” for details). Unfortunately, it was 
impossible to match all of the characteristics considered important on every student; therefore, 
the accuracy of the GPA match was compromised. As a result, the comparison group started 
with a GPA almost 0.4 points higher than the NovaNET students. To account for this 
difference, we have compared the increase in GPA of the NovaNET and comparison groups 
by using a two-way analysis of variance. 

Figure 3 shows GPAs for both groups in spring of 2000 and 2001, as well as the change in 
GPA for both the NovaNET and comparison groups. The WCPSS GPA scale runs from 0 (F) 
to 4 (A), with GPAs above 4 possible due to weighted grades for honors and Advanced 
Placement courses. As illustrated, NovaNET students started out with a C- average while the 
comparison students started out with GPAs closer to a C average. Both groups had a 
significant increase in GPA (the comparison group had a mean difference of .10, p=.0029). 
However, we found that the increase in GPA was not significantly different for the NovaNET 
students than for the comparison students (F=0.51, p>.05). Although the increase in GPA was 



S:\Accountability Team\Kharlow\NovaNET\NovaNET Evaluation.doc 2/26/02 



6 



ERIC 



8 





to 



slightly higher for the NovaNET group than for the comparison group, it was not significantly 
higher. 

Since students in the comparison group were also at-risk for school failure based on their 
characteristics, some may have received services (such as minimum competency assistance or 
after-school tutoring) besides NovaNET which increased their school success. This is one 
possible reason why both NovaNET and the comparison students’ GPAs increased 
significantly from one year to the next. 

Figure 3: Change in Average GPA of NovaNET and Comparison Students 
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Number of Fs 

The next figure illustrates the decrease in number of Fs that NovaNET students achieved 
between the 1999-2000 school year and the 2000-2001 school year during which they 
participated in the NovaNET program. The mean decrease in number of Fs for the NovaNET 
students was .5 (p<.0001), while the comparison group had a slight increase. A two-way 
analysis of variance revealed that this difference was highly significant (p<.0001). The data 
suggest that NovaNET positively impacts the extent to which participants passed their 
courses. 
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Figure 4: Mean Number of Fs Earned by NovaNET and Comparison Students, 2000- 

2001 




The NovaNET program provides students who have fallen behind with the means to make up 
credits, and achieve the ultimate goal of graduating on time. During the 2000-2001 school 
year, the graduation status of 536 NovaNET students, all of whom were participating in 
NovaNET in order to make up credits, was recorded. As shown in Figure 5, almost half of 
students who used NovaNET were on track to graduation at the completion of the NovaNET 
course(s) as of spring 2001 . About one third of the NovaNET students were closer to 
graduation, which means that they were not as far behind in earning credits as when they 
started NovaNET, but are still not on track to graduation. About 20% of the students were no 
closer to graduation than when they began NovaNET, as a result of either failing the class or 
receiving an incomplete. 



Figure 5: On-Track to Graduation: 
NovaNET Students Attempting Credit Recovery, 2001 
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Success by Subject 

The previous NovaNET report (Faircloth & O’Sullivan, 2001) provided information regarding 
the success rate of students for each subject taken in NovaNET since the beginning of the 
SS/HS grant. As shown in Figure 6, the math, social studies and electives courses resulted in 
very high pass rates. Fewer students passed English, and less than half of students taking a 
science NovaNET course passed. The different passing rates for each subject suggests that 
NovaNET may be more appropriately used for some subjects, such as math and social studies, 
than others. In addition, students may be more successful in a difficult subject like science if 
more teacher assistance was available. The nature of the science materials could also be 
reviewed for appropriateness in the context of our NovaNET program. 

Figure 6: NovaNET Passing Rates, By Subject 
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In addition to course pass rates, we calculated the mean increase in overall GPA for students 
enrolled in each subject area to further assess the efficacy of each type of NovaNET course in 
helping students to become more successful (see Figure 7). Students who took a social 
studies course had by far the largest increase in overall GPA. On the other hand, students 
who took an English course in NovaNET had a mean decrease in GPA. NovaNET students 
taking math and social studies students both had a significant increase in GPA. Significance 
was more difficult to achieve for some subjects because of the small number of students 
enrolled. 
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Figure 7: Increase in NovaNET Students’ Overall GPA by Subject 
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Social Studies n=80 
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Elective n=42 


1.49 
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* - Significant at the p< 01 level 
** - Significant at the p<.0001 level 



Suspensions 

Although NovaNET is an academic program, suspension rates were measured to see if the 
availability of NovaNET had behavioral results. As shown in Figure 8, the students who 
participated in NovaNET actually had an increase in number of suspensions and the average 
number of days suspended. The comparison group also showed increases, though not quite as 
large. The NovaNET program, while having a positive effect on students’ academic success, 
did not seem to have an effect on participants’ behavior in 2000-2001. 



Figure 8: Suspensions Data for NovaNET and Comparison Students 
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Dropout Rate 

During the 2000-2001 school year, 3 1 (3%) of the 1,037 NovaNET students with available 
data dropped out. In contrast, 15 (1.5%) of the 995 students in the comparison group dropped 
out, and the overall dropout rate for WCPSS was 2.4%. Thus, the NovaNET group had a 
slightly higher percentage of dropouts in absolute terms. Keep in mind, however, that the 
comparison group was not able to be matched exactly, and represents a slightly higher 
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achieving group than the NovaNET students, and has an even lower dropout rate than WCPSS 
as a whole. 

Figure 9 shows the distribution of NovaNET students across grade levels. In 2000-2001, 
twelfth graders represented the highest percentage of students served in NovaNET (32%), 
followed by ninth graders (29%). Some schools explained that they put a priority on twelfth 
graders in NovaNET since they were closest to graduation. Since WCPSS students are most 
likely to drop out in the 9 th grade (39%), the dropout rate may have been more highly 
impacted if NovaNET was emphasized in the 9 tfi grade (Baenen et al., 2001). In addition, 
measuring the graduation rate of NovaNET students in a few years may give a more accurate 
picture of the effect NovaNET is having on drop out prevention. 

Figure 9: Grade Level of NovaNET Students 
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DISCUSSION 

There are some cautions to keep in mind when interpreting these results. This is the first year 
of full implementation of the NovaNET program, and many schools had implementation 
issues that have not been fully resolved. Educational evaluations indicate that effective 
programs do not always show positive results until the second year. In addition, each school 
uses NovaNET differently. While most students are using NovaNET to make up credits, 
others are using it for remediation or additional practice within a standard class. These 
idiosyncrasies in implementation make generalizations more difficult. 

The inability to match NovaNET students on all variables meant that the comparison group 
was initially somewhat higher achieving than the treatment group. While we controlled for 
these differences statistically, this non-comparability means that we cannot be as confident in 
generalizing our findings. 

The evidence suggests that NovaNET has a positive effect for students academically. It is 
successful in helping students make up credits. Positive results were not demonstrated for 
other areas such as increase in GPA or decrease in rates of suspension. However, both 
students and NovaNET coordinators are enthusiastic about the second chance NovaNET gives 
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to students who may have failed or been unsuccessful in a traditional class (Faircloth & 
O’Sullivan, 2001). 

Questions for consideration by staff include: 

• Why were some students unsuccessful at completing NovaNET courses while others 
were? 

• What can we do to increase the success rates for the NovaNET science curricula, and 
possibly the English curricula? 

• If not, are there other on-line applications or products that might enhance or replace 
NovaNET, especially in areas with lower success rates? 

Issues for further study: 

• Is there a difference in NovaNET success between schools? 

• Does NovaNET have an effect on whether students drop out of school over time? If 
NovaNET coordinators are reporting that NovaNET prevents students from dropping 
out, why do the numbers not reflect that? 

• Is NovaNET effective in competency remediation and course supplementation, or 
should it only be used for course recovery? 

• If resources allow, a future study with a comparison group that better matches the 
NovaNET group may give more robust findings. 



The Safe Schools Healthy Students grant, which provides funding to NovaNET, is ending in 
September, 2002. Although the actual NovaNET program has been funded through the next 
school year, NovaNET coordinators will no longer be grant funded. As a result, schools will 
face the challenge of finding the funds to retain the NovaNET coordinator positions, possibly 
with funding they receive for at-risk students. 
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